Software applications and products can be subjected to various test cases during development and deployment. For example, during bug fixing or feature enhancement, a typical design practice involves running a test plan, including multiple test cases, for the software application. This can help validate the application and make sure the code changes (e.g. the current “build”) did not break any existing functionality. When a change is made to a portion of the software, it can be “committed” to a repository. Upon each commit, the test plan may execute one or more test cases on that build of the software. Moreover, depending on the application being tested, the platform running the test cases may vary. For example, a change to software application code can be deployed on several different devices (e.g., smartphones, tablets, laptops, etc.), which may require running the test cases associated with the build on each device that supports the software application. As a result, testing of large or complicated software applications on different devices can be challenging because it can require execution of numerous test cases that take a long time to execute on a limited number of available devices.
Simply adding additional devices to perform the tests can be problematic because, even with devices that have the same hardware and software configurations, test results (e.g. the amount of time the test took, an amount of memory used, number of processor cycles used, etc.) can vary across test devices. Such test results need to be compared to determine whether the build is broken. However, when this comparison is performed between test results from different devices, differences attributable to how the different devices perform the tests can cause incorrect identifications of malfunctioning or non-malfunctioning builds.
The techniques introduced here may be better understood by referring to the following Detailed Description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements.